Revisiting Label Smoothing Regularization with Knowledge Distillation

Label Smoothing Regularization (LSR) is a widely used tool to generalize classification models by replacing the one-hot ground truth with smoothed labels. Recent research on LSR has increasingly focused correlation between and Knowledge Distillation (KD), which transfers knowledge from teacher model lightweight student penalizing their output’s Kullback–Leibler-divergence. Based this observation, Teacher-free (Tf-KD) method was proposed in previous work. Instead of real model, handcrafted distribution similar guide learning. Tf-KD promising substitute for except its hard-to-tune model-dependent hyperparameters. This paper develops new teacher-free framework LSR-OS-TC, decomposes into two components: Output (OS) Teacher Correction (TC). Firstly, LSR-OS extends KD regime applies softer temperature output softmax layer. smoothing critical stabilizing hyperparameters among different models. Secondly, TC part, larger proportion assigned uniform teacher’s right class provide more informative teacher. The two-component evaluated exhaustively image (dataset CIFAR-100, CIFAR-10, CINIC-10) audio GTZAN) tasks. results showed that can improve performance independently no extra computational cost, especially several deep neural networks where ineffective. further training boost component effectiveness our strategy. Overall, LSR-OS-TC practical substitution be tuned one directly applied other compared original method.